Conventionally, there is an image synthesizing apparatus which synthesizes an image of a virtual object with an image taken by a real camera (for example, Japanese Laid-Open Patent Publication No. 2009-290842, hereinafter, referred to as Patent Document 1). Specifically, in the apparatus described in Patent Document 1, an object in a taken image is detected, and a type and a synthesized state of a virtual object are determined in accordance with the type and the state of the detected object. Then, a synthesized image in which an image of the virtual object is synthesized with the taken image, is generated. When a photographing instruction is provided by a user, the synthesized image is recorded.
However, in Patent Document 1, an image of a virtual object of which a size and an orientation are set previously is merely synthesized with a taken image, and a picture taken by the apparatus of Patent Document 1 does not provide, to the user, a feeling as if the virtual object exists in a real space.
Therefore, an object of the present invention is to provide an image processing technique to take an image which can provide, to a user, a feeling as if a virtual object exists in a real space.
The present invention has the following features to attain the object mentioned above.
One embodiment of the present invention provides a computer-readable storage medium having stored therein an image processing program which is executed by a computer of an image processing apparatus which generates an image in which a virtual object image is superimposed on an image taken by a real camera. The image processing program causes the computer to operate as real image obtaining means, position/orientation information obtaining means, virtual camera setting means, virtual object change means, virtual object image generation means, superimposed image generation means, and storing means. The real image obtaining means obtains a real image of an object which is taken by the real camera. The position/orientation information obtaining means obtains position/orientation information corresponding to a position and an orientation of the real camera in a real space. The virtual camera setting means sets a position and an orientation of a virtual camera in a virtual space on the basis of the position/orientation information obtained by the position/orientation information obtaining means. The virtual object change means changes at least one of a position and an orientation of at least a part of a virtual object located in the virtual space, in accordance with an input performed by a user on input means. The virtual object image generation means generates a virtual object image by taking, by the virtual camera, an image of the virtual object of which the at least one of the position and the orientation has been changed by the virtual object change means. The superimposed image generation means generates a superimposed image in which the virtual object image generated by the virtual object image generation means is superimposed on the real image obtained by the real image obtaining means. The storing means stores the superimposed image in storage means in accordance with a photographing instruction provided by the user.
Note that the position/orientation information, which is obtained by the position/orientation information obtaining means and corresponds to the position and the orientation of the real camera, may be relative position and orientation between a predetermined object existing in the real space and the real camera. In other words, the position/orientation information may be a position and an orientation of the real camera which are based on the predetermined object in the real space, or may be a position and an orientation of the predetermined object in the real space which are based on the real camera. In addition, the position/orientation information may be absolute position and orientation of the real camera which are detected by means for detecting an absolute position (e.g., a GPS) and orientation detection means (e.g., an angular velocity sensor, an acceleration sensor, means for detecting a geomagnetism, or the like).
According to the above, the user can operate the virtual object in the virtual space, and can generate and store a superimposed image in which the operated virtual object is superimposed on a real image. Thus, an image can be taken which can provide, to the user, a feeling as if the virtual object exists in the real space.
In another feature of the present invention, the position/orientation information obtaining means may detect a specific object included in the real image obtained by the real image obtaining means, and may obtain a relative position and a relative orientation of the real camera with respect to the specific object on the basis of a result of the detection.
According to the above, on the basis of the specific object included in the real image taken by the real camera, the relative position and orientation between the specific object in the real space and the real camera can be obtained.
In still another feature of the present invention, the image processing apparatus may be connectable to display means. Until the photographing instruction is provided, the superimposed image generation means repeatedly generates the superimposed image in which the real image obtained by the real image obtaining means and the virtual object image generated by the virtual object image generation means. Then, the image processing program further causes the computer to operate as display control means. The display control means causes the display means to display thereof the superimposed image generated by the superimposed image generation means until the photographing instruction is provided.
According to the above, the superimposed image can be displayed on the display means until the photographing instruction is provided by the user. Thus, the user can confirm the superimposed image which is to be taken (stored).
In still another feature of the present invention, the virtual object may include a first part and a second part. In this case, the virtual object change means includes part change means for changing an orientation of the first part such that the first part of the virtual object is turned to the virtual camera, in accordance with an input performed by the user on the input means.
According to the above, the orientation of the first part can be changed such that the first part of the virtual object is turned to the virtual camera. For example, the face of the virtual object can be turned to the virtual camera, and a feeling as if the virtual object exists in the real space can be provided to the user.
In still another feature of the present invention, when the orientation of the first part is changed such that the first part is turned to the virtual camera, if an orientation of the first part based on the second part is out of a first range, the part change means may set the orientation of the first part to an upper limit or a lower limit of the first range.
According to the above, the orientation of the first part with respect to the second part can be limited within the first range. For example, when the body and the face of the virtual object are defined, the direction of the face can be controlled such that the direction of the face is prevented from being unnatural with respect to the direction of the body.
In still another feature of the present invention, when an input is performed by the user on the input means, the part change means repeatedly may change the orientation of the first part such that the first part of the virtual object is turned to the virtual camera, until the photographing instruction is provided.
According to the above, when the user performs an input, the orientation of the first part is changed such that the first part of the virtual object is always turned to the virtual camera, until the photographing instruction is provided. Thus, a wasteful input can be omitted.
In still another feature of the present invention, the real camera may be a stereo camera. The real image obtaining means obtains a real image for a left eye, which is taken by a real camera for a left eye of the stereo camera, and a real image for a right eye, which is taken by a real camera for a right eye of the stereo camera. The position/orientation information obtaining means obtains position/orientation information corresponding to a position and an orientation of each of the real camera for left eye and the real camera for a right eye of the stereo camera. The virtual camera setting means sets a position and an orientation of a left virtual camera in accordance with the position/orientation information of the real camera for a left eye which is obtained by the position/orientation information obtaining means, and sets a position and an orientation of a right virtual camera in accordance with the position/orientation information of the real camera for a right eye which is obtained by the position/orientation information obtaining means. The part change means changes the orientation of the first part such that the first part of the virtual object is turned to a midpoint between the left virtual camera and the right virtual camera. The virtual object image generation means generates a virtual object image for a left eye and a virtual object image for a right eye by taking, by the left virtual camera and the right virtual camera, images of the virtual object of which the orientation of the first part has been changed by the part change means. The superimposed image generation means generates a superimposed image for a left eye in which the virtual object image for a left eye is superimposed on the real image for a left eye, and a superimposed image for a right eye in which the virtual object image for a right eye is superimposed on the real image for a right eye.
According to the above, a stereoscopic image in which the virtual object is superimposed on a taken image of the real space can be generated. In this case, the first part of the virtual object can be turned to the left and right virtual cameras. Thus, the user can taken a stereoscopic image, and can obtain enhanced augmented reality.
In still another feature of the present invention, the virtual object change means may include moving means for moving the virtual object in accordance with an input performed by the user on the input means.
According to the above, the user can move the virtual object in the virtual space, and can obtain a feeling as if the virtual object moves in the real space.
In still another feature of the present invention, the virtual object change means may include moving means for moving, in accordance with an input performed by the user on the input means, the virtual object such that a moving direction of the virtual object on the display means is the same as a direction of the input.
According to the above, the direction of the input on the input means agrees with the moving direction in a display. Thus, the user can move the virtual object in a desired direction by an intuitive operation.
In still another feature of the present invention, if a position of the virtual object is out of a second range in the virtual space when the virtual object is moved in accordance with the input performed by the user on the input means, the moving means may correct the position of the virtual object to be within the second range.
According to the above, a movement range of the virtual object can be limited.
In still another feature of the present invention, the virtual object change means may include pose change means for changing a pose of the virtual object in accordance with an input performed by the user on the input means.
According to the above, the user can change the pose of the virtual object, and can obtain a feeling as if the virtual object exists in the real space.
In still another feature of the present invention, the image processing program may further cause the computer to operate as selection means. The selection means selects one virtual object from among a plurality of virtual objects located in the virtual space. Then, the virtual object change means changes at least one of a position and an orientation of at least a part of the virtual object selected by the selection means, in accordance with an input performed by the user on the input means.
According to the above, the user can select a virtual object and can change the position or the orientation of at least a part of the selected virtual object.
In still another feature of the present invention, the image processing program may further cause the computer to operate as selection means. The selection means selects one virtual object from among a plurality of virtual objects located in the virtual space. The superimposed image generation means generates a superimposed image in which a cursor indicating that the virtual object is selected by the selection means is further superimposed. The virtual object change means changes at least one of a position and an orientation of at least a part of the virtual object selected by the selection means, in accordance with an input performed by the user on the input means. In accordance with a photographing instruction provided by the user, the storing means sets the cursor to be non-displayed and stores the superimposed image in the storage means.
According to the above, until the superimposed image is stored in accordance with the photographing instruction, the user can select and change a virtual object while confirming a superimposed image displayed on the display means. In addition, since the cursor is superimposed and displayed on the display means, the user can confirm the selected virtual object. Moreover, when the superimposed image is stored, the cursor can be set to be non-displayed, and a picture can be taken in which the virtual object appears as if existing in the real space.
In another embodiment, an image processing apparatus implementing the above respective means may be configured. In still another embodiment, a plurality of elements implementing the above respective means may be configured as one image processing system by mutually operating. The image processing system may be constituted of one apparatus or a plurality of apparatuses.
According to the present invention, an image can be taken which provides, to the user, a feeling as if the virtual object exists in the real space.
These and other objects, features, aspects and advantages of the present invention will become more apparent from the following detailed description of the present invention when taken in conjunction with the accompanying drawings.